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We present a search for flavor changing neutral currents in decays of top quarks. The analysis is 
based on a search for ti — > £'is££+]ets (£, I' — e, final states using 4.1 fb -1 of integrated luminosity 
of pp collisions at y/s — 1.96 TeV. We extract limits on the branching ratio B(t —t Zq) (q = u,c 
quarks), assuming anomalous tuZ or tcZ couplings. We do not observe any sign of such anomalous 
coupling and set a limit of B < 3.2% at 95% C.L. 
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Flavor changing neutral currents (FCNC) allow for 
transitions between quarks of different flavor but same 
electric charge. FCNC are sensitive indicators of physics 
beyond the standard model (BSM) , because they are sup- 
pressed in the standard model (SM). 

In this paper, we search for FCNC decays of the top 
(t) quark [y. Within the SM the top quark decays 
into a W boson and a b quark with a rate propor- 
tional to the Cabibbo-Kobayashi-Maskawa (CKM) ma- 
trix element squared, |14t>| 2 Q- Under the assumption 
of three fermion families and a unitary 3x3 CKM ma- 
trix, the | Vtb | element is severely constrained to |14t>| = 
0.999152±°;°°°^° 0. While the SM branching frac- 
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tion for t — > Zq (q = u,c quarks) is predicted to be 
w 10~ 14 supersymmetric extensions of the SM with 
or without i?-parity violation, or quark compositeness 
predict branching fractions as high as w 10~ 4 043- The 
observation of the FCNC decay t — > Zq would therefore 
provide evidence of contributions from BSM physics. 

We analyze top-pair production (it), where either one 
or both of the top quarks decay via t — ¥ Zq or their charge 
conjugates (hereafter implied). Any top quark that does 
not decay via t — > Zq is assumed to decay via t — > Wb. 
We assume that the t — ► Zq decay is generated by an 
anomalous FCNC term added to the SM Lagrangian 

•Cfcnc = 

2 Si neJco S e H .- tlMiZ-atqZls) qZ^ + h.c, (1) 

where q, t, and Z are the quantum fields for up or charm 
quarks, top quarks, and for the Z boson, respectively, 
e is the electric charge, and 8w the Weinberg angle. 
We thereby introduce dimcnsion-4 vector, Vt q z, and axial 
vector, a tq z, couplings as defined in @. We find in @, ll| 
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Figure 1: Lowest-order diagram for FCNC it — >• WbZq' pro- 
duction, where q' can be either a u or c quark, and the 
and Z bosons decay leptonically. 



that the next-to- leading order (NLO) effects due to per- 
turbativc QCD corrections are negligible when extracting 
the branching ratio limits to the leading order (LO) in 
Eq.[B 

We investigate channels where the W and Z bosons 
decay leptonically, as shown in Fig. [TJ The u, c, and 
b quarks subsequently hadronizc, giving rise to a final 
state with three charged leptons (£ = e, /i), an imbalance 
in momentum transverse to the pp collision axis (^ T , 
assumed to be from the escaping neutrino in the W — > tv 
decay), and jets. 

This is the first search for FCNC in ti decays with 
trilcpton final states. This mode provides a distinct sig- 
nature with low background, albeit at the cost of sta- 
tistical power. The first measurement (b — > sj) was 
published in 1995 by the CLEO Collaboration Nu- 
merous studies have been done since then to search 
for FCNC processes in meson decays, i.e., b —> Zs in 
B+ -> K*+e+£- [IM1, B -> K*vv [llLand B s . d -> 
£+£- [H [H or s -> Zd in K+ -> tt+vF^. Using the 
D + — > 7r + /i + /i~ final state in 1.3 fb _1 of integrated lu- 
minosity, the DO Collaboration has set the best branch- 
ing ratio (B) limits on the FCNC c — > Zu process at 
B{D+ 7r+/i+M~) <3.9xl0" 6 at 90% C.L. [l7|. There 
are theoretical arguments as to why top quark decays 
may be the best way to study flavor violating couplings of 
mass-dependent interactions [HI, [l!| . FCNC tqZ and tq^y 
couplings have been studied by the CERN e + e~ Collider 
(LEP), DESY ep Collider (HERAL and Fermilab pp Col- 
lider (Tevatron) experiments [2(3-[24j]. The DO Collabora- 
tion has recently published limits on the branching ratios 
determined from FCNC gluon-quark couplings using sin- 
gle top quark final states [25|. The 95% C.L. upper limit 
on the branching ratio of t —> Zq from the CDF Collab- 
oration uses 1.9 fb _1 of integrated luminosity, assumes a 
top quark mass of m t = 175 GeV and uses the measured 
cross section of a t i = 8.8 ± 1.1 pb [24|. This result ex- 
cludes branching ratios of B(t — > Zq) > 3.7%, with an 
expected limit of 5.0% ± 2.2%. To obtain these results, 
CDF exploited the two lepton plus four jet final state. 
This signature occurs when one of the pair-produced top 



quarks decays via FCNC to Zq, followed by the decay 
Z — > ee or Z — > /u/i. The other top quark decays to 
Wb, followed by the hadronic decay of the W boson. 
This dilepton signature suffers from large background, 
but profits from more events relative to the trilepton fi- 
nal states investigated in this Letter. 

This analysis is based on the measurement of the 
W Z production cross section in Ivll final states [26| 
using 4.1 fb _1 of integrated luminosity of pp collisions 
at *Js = 1.96 TcV. We extend the selection by ana- 
lyzing events with any number of jets in the final state 
and investigate observables that are sensitive to the sig- 
nal topology in order to select events with WZ —> Ivil 
decays that originate from the pair production of top 
quarks. 

A detailed description of the DO detector can be found 
elsewhere [27| . Here, we give a brief overview of the main 
sub-systems of the detector. The innermost part is a 
central tracking system surrounded by a 1.9 T supercon- 
ducting solenoidal magnet. The two components of the 
central tracking system, a silicon microstrip tracker and 
a central fiber tracker, are used to reconstruct pp interac- 
tion vertices and provide the measurement of the momen- 
tum of charged particles within the pseudorapidity range 
of 1 77 1 < 2 (with 77 defined relative to the center of the 
detector). The tracking system and magnet are followed 
by the calorimetry that consists of central (CC) and end 
(EC) electromagnetic and hadronic uranium-liquid ar- 
gon sampling calorimeters, and an intercryostat detector 
(ICD). The CC and two EC calorimeters cover \r\\ < 1.1 
and 1.5 < |?/| < 4.2, respectively, while the ICD cov- 
ers 1.1 < \r)\ < 1.4. The calorimeter measures energies 
of hadrons, electrons, and photons. The muon system 
consists of a layer of drift tubes and scintillation coun- 
ters inside a 1.8 T toroidal magnet. Two similar layers 
are outside of the toriodal magnet and the entire system 
covers \r)\ < 2. 

An electron is identified from the properties of clusters 
of energy deposited in the CC, EC, or ICD that match 
a track reconstructed in the central tracker. Because of 
the lack of far forward coverage of the tracker, we define 
EC electrons only within 1.5 < < 2.5. The calorime- 
ter clusters in the CC and EC are required to pass the 
isolation cut 

E tot (AK < 0.4) - E EM (ATZ < 0.2) 
E E m{AK<0.2) 

for "loose" electrons and < 0.07 for "tight" electrons, 
where E tot is the total energy in the EM and hadronic 
calorimeters, -Eem is the ener gy found in t he EM 
calorimeter only, and A1Z = \J (A^) 2 + (A77) 2 , where 
4> is the azimuthal angle. For the intercryostat region 
(ICR), 1.1 < 1 77 1 < 1.5, we form clusters from the energy 
deposits in the CC, ICD, or EC detectors. These clusters 
arc identified as electrons if they pass a neural network 
requirement that is based on the characteristics of the 
shower and associated track information. A muon can- 
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didatc is reconstructed from track segments within the 
muon system that are matched to a track reconstructed 
in the central tracker. The trajectory of the muon can- 
didate must be isolated from other tracks within a cone 
of A1Z < 0.5, with the sum of the tracks' transverse mo- 
menta, pt, in a cone less than 4.0 GeV for "loose" muons 
and less than 2.5 GeV for "tight" muons. Tight muon 
candidates must also have less than 2.5 GeV of calorime- 
ter energy in an annulus of 0.1 < A1Z < 0.4. Jets are 
reconstructed from the energy deposited in the CC and 
EC calorimeters, using the "Run II midpoint cone" algo- 
rithm [HI of size ATI = 0.5, within \r]\ < 2.5. 

Monte Carlo (MC) samples of WZ and ZZ back- 
ground events are produced using the pythia genera- 
tor [2j| . The production of the W and Z bosons in asso- 
ciation with jets (IV+jets, -Z+jets), collectively referred 
to as V+jets, as well as tt processes are generated using 
ALPGEN [30[ interfaced with pythia for parton evolution 
and hadronization. In all samples the CTEQ6L1 par- 
ton distribution function (PDF) set is used, along with 
m t = 172.5 GeV. The tt cross section is set to the SM 
value at this top quark mass, i.e 



of B = T(t Zq)/T toU where Ttot contains t 
t — > Zq decays only, can be written as 



Wb and 



o-ti 



7.46t°f 7 pb[3]J 



This uncertainty is mainly due to the scale dependence, 
PDFs, and the experimental uncertainty on m t [32j]. 

All MC samples are passed through a GEANT [33j sim- 
ulation of the DO detector and overlaid with data events 
from random beam crossings to account for the underly- 
ing event. The samples are then corrected for the lumi- 
nosity dependence of the trigger, reconstruction efficien- 
cies in data, and the beam position. All MC samples are 
normalized to the luminosity in data using NLO calcula- 
tions of the cross sections, and are subject to the same 
selection criteria as applied to data. 

The signal process is generated using the pythia gen- 
erator with the decay t — >• Zq added. The Z boson 
helicity is implemented by rewcighting an angular dis- 
tribution of the positively charged lepton in the decay 
t ->• Zq -> i + frq using CompHEP [Hj], modified by 
the addition of the Lagrangian of Eq. [TJ The variable 
cosO* used for the reweighting is defined by the angle 
6* between the Z boson's momentum in the top quark 
rest frame and the momentum of the positively charged 
lepton in the Z boson rest frame. We assume in the 
analysis that the vector and axial vector couplings, as 
introduced in Eq. [TJ are identical to the correspond- 
ing couplings for neutral currents (NC) in the SM, i.e., 
1/2 -4/3 sin 2 9 W = 0.192 and a tuZ = 1/2, where 
= 0.231. To study the influence of different val- 
ues of the couplings, we also analyse the following cases: 
(i.a) v tu z = l,a tuZ = 0; (i.b) v tu z = 0, a tu z = 1; and 
(ii) vtuz = o-tuz = 1/ v2- As expected, the first two give 
identical results. The difference obtained by using cases 
(i), (ii), and using the values of the SM NC couplings 
is included as systematic uncertainty. Therefore, our re- 
sult is independent of the actual values of Vt q z and at q z- 
Since we do not distinguish c and u quark jets our results 
are valid also for u and c quarks separately. 

The total selection efficiency, calculated as a function 



VtuZ = 

sin 2 9w 



e tt — (1 - B) 2 ■ e-ti^w+bw-b ( 2 ) 

+ 2B(1 - B) ■ e t t^ZqW-b + B 2 ■ (-ti^ZqZq, 



where the efficiency e t i 



-z for the SM tt back- 



tt^W+bW-b 

ground contribution is used, along with the efficiencies 
e ti^ZqW-T> an d tti^,ZqZq that include the FCNC top 
quark decays. 

We consider four independent decay signatures: eee + 
$ T + X, eep + $ T + X, fj,/j,e + $T + X , and mjlh + $t+X, 
where X is any number of jets. We require the events to 
have at least three lepton candidates with pr > 15 GeV 
that originate from the same pp interaction vertex and 
are separated from each other by A1Z > 0.5. Jets are ex- 
cluded from consideration unless they have pr > 20 GeV. 
We also require that the jets are separated from electrons 
by A7Z > 0.5. There is no fixed separation cut between 
the muon and jets but the muon isolation requirement 
rejects most muons within A1Z < 0.4 of a jet. The event 
must also have ~$ T > 20 GeV, which is calculated from 
the energy found in the calorimeter cells and pr corrected 
for any muons reconstucted in the event. Furthermore, 
all energy corrections applied to electrons and jets are 
propagated through to the $ T . 

Events are selected using triggers based on electrons 
and muons. There are several high-pr leptons from the 
decay of the heavy gauge bosons providing a total trigger 
efficiency for all signatures of 98% ± 2%. 

To identify the leptons from the Z boson decay, we 
consider only pairs of electrons or muons, additionally 
requiring them to have opposite electric charges. If no 
lepton pair is found within the invariant mass intervals 
of 74-108 GeV (ee), 65-115 GeV {pp) or 60-120 GeV 
(ee, with one electron in the ICR) the event is rejected, 
else, the pair that has an invariant mass closest to the Z 
boson mass Mz is selected as the Z boson. The lepton 
with the highest pt of the remaining muons or CC/EC 
electrons in the event is selected as originating from the 
W boson decay. From simulation, this assignment of the 
three charged leptons to Z and W bosons is found to be 

100% correct for eep and pp,e, and about 92% and 89% 
for the eee and p/ifi channels, respectively. 

Thresholds in the selection criteria are the same as 
in Ref. [26| and the acceptance multiplied by efficiency 
results are summarized in Table [TJ for the FCNC signal. 
These values are calculated with respect to the total rate 
expected for all three generations of leptonic W and Z 
decays. 

In addition to SM WZ production, the other major 
background is from processes with a Z boson and an 
additional object misidentificd as the lepton from the W 
boson decay (e.g., from Z+jets, ZZ, and Z7). A small 
background contribution is expected from processes such 
as VF+jets and SM tt production. The WZ, ZZ, and tt 
backgrounds are estimated from the simulation, while the 
V+jets and Z"f backgrounds are estimated using data- 
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nipt 
j 


Inclusive 


1 


> 2 


Channel 


C tt^ZqW-b (%) 


eee 


1.65 ±0.24 


(7.65 ± 1.45) • 10"- 3 


0.57 ±0.09 


1.00 ±0.15 


ee\x 


1.92 ±0.18 


(6.77 ± 1.05) • 10" 2 


0.58 ±0.06 


1.17± 0.11 


Line 


1.23 ± 0.13 


(3.37 ±0.73) • 10-* 


0.34 ±0.04 


0.84 ±0.10 




1.48 ±0.19 


(3.05 ±0.74) • 10" 2 


0.38 ±0.06 


1.06 ±0.15 


Channel 


ttt^ZqZq (%) 


eee 


1.22 ±0.18 


(4.69 ±0.68) • 10" 2 


0.41 ±0.06 


0.76 ±0.11 


eep, 


3.75 ±0.38 


(1.07 ±0.11) • 10- 1 


1.08 ±0.11 


2.56 ± 0.25 


jj,/j,e 


1.47 ± 0.16 


(3.22 ±0.57) • 10~ 2 


0.38 ±0.05 


1.06 ±0.32 


fJLfXfJt 


2.76 ±0.36 


(3.63 ±0.69) • lO" 2 


0.63 ±0.09 


2.10 ±0.28 



Table I: Final efficiencies in % including detector and kinematic acceptance as well as detector efficiencies for each decay 
signature as a function of jet multiplicity 7ij ot . The efficiency, e, is defined assuming fully leptonic decays of the vector bosons 
from top quarks, as defined as in Eq. [2] The statistical and systematic uncertainties have been added in quadradure. 



driven methods. 

One or more jets in V+jets events can be misidenti- 
fied as a lepton from W or Z boson decays. To estimate 
this contribution, we define a false lepton category for 
electrons and muons. A false electron is required to have 
most of its energy deposited in the electromagnetic part 
of the calorimeter and satisfy calorimeter isolation cri- 
teria for electrons, but have a shower shape inconsistent 
with that of an electron. A muon candidate is categorized 
as false if it fails isolation criteria, as determined from 
the total pt of tracks located within a cone AT?. = 0.5 
around the muon. These requirements ensure that the 
false lepton is cither a misidentified jet or a lepton from 
the semi-leptonic decay of a heavy-flavor quark. Using 
a sample of data events, collected using jet triggers with 
no lepton requirement, we measure the ratio of misidenti- 
fied leptons passing two different selection criteria, false 
lepton and signal lepton, as a function of pt in three 
bins, 7jj 0t = 0, 1, and > 2, where nj et is the number of 
jets. We then select a sample of Z boson decays with 
at least one additional false lepton candidate for each fi- 
nal state signature. The contribution from the V±jets 
background is estimated by scaling the number of events 
in this Z+false lepton sample by the corresponding px- 
dependent misidentification ratio. 

Initial or final state radiation in Z~f events can mimic 
the signal process if the photon either converts into an 
e + e _ pair or is wrongly matched with a central track 
mimicking an electron and the $ T is mismeasured. As a 
result the Zj process is a background to the final state 
signatures with W — > ev decays. To estimate the contri- 
bution from this background, we model the kinematics 
of these events using the Z^ NLO MC simulation (35| . 
We scale this result by the rate at which a photon is 
misidentified as an electron. This rate is obtained using 
a data sample of Z — > fip events containing a radiated 
photon, as it offers an almost background-free source of 
photons. The invariant mass M(pfij) is reconstructed 
and required to be consistent with the Z boson mass. 
The Z — ¥ fip, decay is chosen to avoid any ambiguity 
when assigning the electromagnetic shower to the final 



state photon candidate. As the Z7 NLO MC does not 
model recoil jets, PYTHIA MC samples are used to esti- 
mate Z'y background jet multiplicities and $ T . As the 
pythia samples do not contain events with final state 
radiation, we find the fraction of Z7 events in data and 
pythia MC that pass our ~$ T cut and take the difference 
as a systematic uncertainty. 

After all selection criteria have been applied, we ob- 
serve a total of 35 candidate events and expect 31.7 ± 
0.3(stat) ± 3.9(syst) background events from SM pro- 
cesses. The statistical uncertainty is due to MC statis- 
tics while the sources of systematic uncertainties arc dis- 
cussed later. Table [TT] summarizes the number of events 
in each nj e t bin. The observed number of candidate and 
background events for each topology, summing over nj e t, 
are summarized in Table IIIII In Tables [TT] and Mil and in 
all the following figures, we assume a B of 5%. 

To achieve better separation between signal and back- 
ground, we analyze the nj e t and Ht distributions (defined 
as the scalar sum of transverse momenta of all leptons, 
jets, and J^t)i an d the reconstructed invariant mass for 
the products of the decay t — > Zq. 

The jet multiplicities in data, SM background, and in 
FCNC top quark decays arc shown in Fig. H FCNC it 
production leads to larger jet multiplicities and also a 
larger Ht- This is shown in Fig. [3] 

To further increase our sensitivity we reconstruct the 
mass of the top quark that decays via FCNC to a Z 
boson and a quark (t — > Zq). In events with ?ij Ct = 0, 
this variable is not defined. In events with one jet, we 
calculate the invariant mass, m[ oco = M(Z, jet), from 
the 4-momenta of the jet and the identified Z boson, to 
reconstruct rut- For events with two or more jets, we use 
the jet that gives a m\ ec ° closest to m t — 172.5 GeV. The 
m\ cco distribution is shown in Fig.Uta). In Fig.|U[b), we 
present a 2-dimensional distribution of m\ eco and Ht- 

None of the observables in Figs.[5]~|3]show evidence for 
the presence of FCNC in the decay of ti. We therefore set 
95% C.L. limits on the branching ratio B(t — > Zq). The 
limits are derived from 10 bins of the Ht distributions 
for njet = 0, 1, and > 2. For the channels with nj e t = 1 
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n jct 





1 


> 2 


Background 


25.66 ±0.28 ±3.26 


5.06 ±0.14 ±0.56 


0.92 ±0.08 ±0.09 


tt -> WbZq 


0.20 ±0.03 


1.80 ±0.27 


3.87 ±0.56 


tt — > ZqZq 


0.002 ±0.001 


0.020 ±0.003 


0.050 ±0.007 


Observed 


30 


4 


1 



Table II: Number of observed events, expected number of ti FCNC events, and number of expected background events for each 
rij e t bin with statistical and systematic uncertainties. The MC statistical uncertainty on the tt signal is negligible, and we only 
present the systematic uncertainties. We assume B = 5%. 



Source 


eee 


ee/x 


e/x/x 




WZ 
ZZ 

V + jets 
Zj 

tt^t WbWb 


6.64 ± 0.07 ± 1.19 
0.33 ±0.03 ±0.06 
0.60 ±0.13 ±0.11 
0.18 ±0.05 ±0.08 
0.04 ±0.01 ±0.01 


7.51 ±0.08 ±1.11 
1.76 ±0.07 ±0.17 
0.40 ±0.18 ±0.17 

< 0.001 
0.04 ±0.01 ±0.01 


4.75 ±0.06 ±0.69 
0.46 ±0.04 ±0.07 
0.48 ±0.10 ±0.01 
0.66 ±0.07 ±0.38 
0.05 ±0.01 ±0.01 


6.10 ± 0.07 ± 1.00 
1.30 ±0.06 ±0.21 
0.22 ± 0.05 ± 0.03 

< 0.001 
0.04 ±0.01 ±0.01 


Background 


7.89 ± 0.16 ± 1.20 


9.71 ± 0.21 ± 1.14 


6.40 ±0.14 ±0.79 


7.66 ± 0.11 ± 1.02 


tt -> WbZq 
tt — > ZqZq 


1.57 ±0.22 
0.010 ±0.001 


1.73 ±0.17 
0.029 ±0.003 


1.17 ±0.13 
0.011 ±0.001 


1.41 ±0.18 
0.022 ±0.003 


Observed 


8 


13 


9 


5 



Table III: Number of observed events, expected number of tt FCNC events, and number of expected background events for each 
final state signature with statistical and systematic uncertainties. The MC statistical uncertainty on the tt signal is negligible, 
and we only present the systematic uncertainties. We assume B — 5%. 

172.5 GcV to 175 GcV in ti MC samples with the dif- 
ference in the result taken as a systematic uncertainty. 
We vary the Vt q z and at q z couplings as explained be- 
fore Eq. [21 resulting in a 1% systematic uncertainty on 
the acceptance. Due to the uncertainty on the theoret- 
ical cross sections for WZ and ZZ production, we as- 
sign a 10% [36| systematic uncertainty to each. The ma- 
jor sources of systematic uncertainty on the estimated 
y±jets contribution arise from the $ T requirement and 
the statistics in the multijet sample used to measure 
the lepton-misidentification rates. These effects are es- 
timated independently for each signature and found to 
be between 20% and 30%. The systematic uncertainty 
on the Z^f background is estimated to be 40% and 58% 
for the eee and /x/xe channels, respectively Uncertain- 
ties on jet energy scale, jet energy resolution, jet recon- 
struction, and identification efficiency are estimated by 
varying parameters within their experimental uncertain- 
ties. For njet = the uncertainty is found to be 1%, for 
njet = 1 it is 5%, and for n,j e t > 2 it is 20%. The measured 
integrated luminosity has an uncertainty of 6.1% [37l |. 

We use a modified frequentist approach [38[ where the 
signal confidence level CL S , defined as the ratio of the 
confidence level for the signal±background hypothesis to 
the background-only hypothesis (CL S = CL s+ t,/CLb), is 
calculated by integrating the distributions of a test statis- 
tic over the outcomes of pseudo-experiments generated 
according to Poisson statistics for the signal+background 
and background-only hypotheses. The test statistic is 
calculated as a joint log-likelihood ratio (LLR) obtained 
by summing LLR values over the bins of the ifr distri- 
butions. Systematic uncertainties are incorporated via 



• i-H 



4) 



a 

> 



30 



20- 



tf FCNC, B(t->Zq)=5% 
— ■ — Data 
1 | WZ 

p-| zz D0,4.1fb" ! 

I I Other backgrounds 




Figure 2: Distribution of 7ij 8 t for data, for simulated FCNC 
ti signal, and for the expected background. The ZqZq signal 
is included in the tt FCNC contribution but is expected to be 
small, as can be seen from Tables HTl and ITTT1 



and n 
in m 



jet 
rcco 
t ! 
-.reco 

h 



> 2, we split each distribution into 4 bins 



< 120 GeV, 



< ml cco < 200 GcV, and 



< 150 GeV, 
> 200 GeV. 



120 < m\ eco 



150 



When calculating the limit on the branching ratio we 
consider several sources of systematic uncertainty. The 
systematic uncertainties for lepton-identification efficien- 
cies are 15% (eee), 11% (ee/x), 9% (/x/xe), and 12% (/x/x/j). 
The systematic uncertainty assigned to the choice of PDF 
is 5%. In addition, we assign 9% systematic uncertainty 
on <j t i [3l|. This includes the dependence on the un- 
certainty of m t 321 1 . Furthermore, m t is changed from 
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Figure 3: Ht distribution of data, FCNC ti signal, and ex- 
pected background for events with (a) nj e t = 0, (b) nj e t = 1, 
and (c) rijet > 2. 



Gaussian smearing of Poisson probabilities for signal and 
backgrounds in the pseudo-experiments. All correlations 
between signal and backgrounds are maintained. To re- 
duce the impact of systematic uncertainties on the sen- 
sitivity of the analysis, the individual signal and back- 
ground contributions are fitted to the data, by allowing 
a variation of the background (or signal+background) 
prediction, within its systematic uncertainties [391 ] . The 
likelihood is constructed via a joint Poisson probability 
over the number of bins in the calculation, and is a func- 
tion of scaling factors for the systematic uncertainties, 



Figure 4: (a) m\ cc ° distribution of data, FCNC ti signal, and 
expected background for events with n,- e t > 1; (b) m\ cc ° vs. 
Ht distribution of data, FCNC ti signal, and background for 
events with 7ij et > 1. 



which are given as Gaussian constraints associated with 
their priors. 

We determine an observed limit of B(t —> Zq) < 3.2%, 
with an expected limit of < 3.8% at the 95% C.L. The 
limits on the branching ratio are converted to limits at 
the 95% C.L. on the FCNC vector, v tq z, and axial vec- 
tor, a tq z, couplings as defined in Eq. Q] using the rela- 
tion given in [6j. This can be done for any point in the 
{vtqZ, atqz) parameter space and for different quark fla- 
vors (it, c) since the differences in the helicity structure 
of the couplings are covered as systematic uncertainties 
in the limit on the branching ratio. Assuming only one 
non- vanishing vt q z coupling (a tq z = 0), we derive an 
observed (expected) limit of v tq z < 0.19 (< 0.21) for 
m t = 172.5 GeV. Likewise, this limit holds assuming 
only one non-vanishing at q z coupling. Figure [3] shows 
current limits from experiments at the LEP, HERA, and 
Tevatron colliders as a function of the FCNC couplings 
K tul (defined in Ref. [(|) and Vt u z for mt = 175 GeV. 

In summary, we have presented a search for top quark 
decays via FCNC in ti events leading to final states in- 
volving three leptons, an imbalance in transverse momen- 
tum, and jets. These final states have been explored for 
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the first time in the context of FCNC couplings. In the 
absence of signal, we expect a limit of B(t —> Zq) < 3.8% 
and set a limit of B(t ->• Zq) < 3.2% at the 95% C.L. 
which is currently the world's best limit. This trans- 
lates into an observed limit on the FCNC coupling of 
v tq z < 0.19 for m t = 172.5 GeV. 



tuyl 



Figure 5: Upper limits at the 95% C.L. on the anomalous 
Ktu-y and vtuz couplings assuming m t — 175 GeV. Both DO 
and CDF limits on v tq z are scaled to the SM cross section 
of a t i = 6.90 pb [3l|. Anomalous axial vector couplings and 
couplings of the charm quark are neglected: at u z = VtcZ = 
atcZ — Ktc-y = 0. The scale parameter for the anomalous 
dimension-5 coupling ittui is set to A = mt = 175 GeV [2lj . 
Any dependence of the Tevatron limits on Kt„ 7 is not dis- 
played as the change is small and at most 6% for K tU7 = 0.5. 
The domain excluded by DO is represented by the light (blue) 
shaded area. The hatched area corresponds to the additional 
domain excluded at HERA by the HI experiment [21| . Also 
shown are upper limits obtained at LEP by the L3 exper- 
iment 12011 (green dashed), at HERA by the ZEUS experi- 
ment [22] ( grey dashed), and at the Tevatron by the CDF 
experiment [23L l24| (magenta dashed). The region above or 
to the right of the respective lines is excluded. 
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